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With the advent of large-scale quantum annealing devices, several challenges have emerged. For 
example, it has been shown that the performance of a device can be significantly affected by several 
degrees of freedom when programming the device; a common example being gauge selection. To 
date, no experimentally-tested strategy exists to select the best programming specifications. We 
developed a score function that can be calculated from a number of readouts much smaller than the 
number of readouts required to find the desired solution. We show how this performance estimator 
can be used to guide, for example, the selection of the optimal gauges out of a pool of random gauge 
candidates and how to select the values of parameters for which we have no a priori knowledge 
of the optimal value. For the latter, we illustrate the concept by applying the score function to 
set the strength of the parameter intended to enforce the embedding of the logical graph into 
the hardware architecture, a challenge frequently encountered in the implementation of real-world 
problem instances. Since the harder the problem instances, the more useful the strategies proposed 
in this work are, we expect the programming strategies proposed to significantly reduce the time of 
future benchmark studies and in help finding the solution of hard-to-solve real-world applications 
implemented in the next generation of quantum annealing devices. 


I. INTRODUCTION 

The fabrication of scalable hardware architectures for 
quantum annealers mm to solve discrete optimization 
problems has sparked interest in quantum annealing al¬ 
gorithms ism]. Current research studies focus on both 
fundamental and practical important questions, includ¬ 
ing the implementation of real-world applications 
defining criteria for detecting quantum speedup and the 
computational role of quantum tunneling [10, [Tl], pro¬ 
posals for error-supression schemes [12], benchmark stud¬ 
ies comparing classical and quantum annealing EM, 
and using spin-glass perspectives into the hardness of 
computational problems studied [13 m]. 

The next generation of quantum annealers will likely 
allow for the exploration of harder and more interesting 
problems instances. Even in the case of a quantum pro¬ 
cessor with only ^500 qubits, one could already see the 
appearance of some hard to solve instances, for which 
the optimal solution was not found out of a few thou¬ 
sand annealing cycles m ■ It is precisely for these hard 
instances that the methods developed in this paper are 
the most useful, since they allow us to extract the best 
programming settings enhancing the probability of find¬ 
ing the ground state, therefore reducing significantly the 
time to solution for both future benchmark studies and 
for real-world applications. 

The first step of solving a problem using a quan¬ 
tum annealer is to map the problem to the hardware 
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architecture. The quantum hardware employed con¬ 
sists of 64 units of a recently characterized eight-qubit 
unit cell [2 123- Post-fabrication characterization de¬ 
termined that only 509 qubits out of the 512 qubit ar¬ 
ray can be reliably used for computation (Fig. [4] in Ap¬ 
pendix 0 . The array of coupled superconducting flux 
qubits is, effectively, an artificial Ising spin system with 
programmable spin-spin couplings and transverse mag¬ 
netic fields. It is designed to solve instances of the follow¬ 
ing (NP-hard [23] ) classical optimization problem: Given 
a set of local longitudinal {hi} and an interaction matrix 
{Jij}, find the assignment s* = that mini¬ 

mizes the objective function E( s), 

Ei s i n g(s) = ^ ^ h{Si T ^ ^ J{jSiSj , (1) 

l<i<N 1 <i<j<N 

where, \hi\ < 2, \Jij\ < 1, and Si G {+1,-1}. Finding 
the optimal s* is equivalent to finding the ground state 
of the corresponding Ising Hamiltonian, H p = JA /i^crf + 
Yi<j where erf are Pauli matrices acting on the 

ith spin. 

Physical realizations of quantum annealing come with 
certain degrees of freedom affecting the performance of 
the quantum annealing device. Each realization of such 
degrees of freedom determine a unique Hamiltonian spec¬ 
ification or realization of Eq. E Although there is some 
understanding of the several factors affecting the perfor¬ 
mance of quantum annealing devices [ 23 ] , there is a need 
for concrete scalable strategies coping with the analog 
control error (ACE) intrinsic to physical hardware imple¬ 
mentations. For example, to the best of our knowledge 
there is no known “rule-of-thumb” in the selection of such 
parameters, thus motivating our study, (we rule out the 
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only one wide spread in the community, in Appendix IB}) 

In the absence of noise or any miscalibration, the per¬ 
formance of the quantum device should be the same un¬ 
der any gauge realization El- Previous studies show 
that the current generation of D-Wave devices with hun¬ 
dreds of qubits is very sensitive to this selection mini. 
Some other degrees of freedom correspond to parameters 
we do not yet know a priori how to set. This is the case 
for penalty strength in the construction of quadratic un¬ 
constrained binary optimization (QUBO) Hamiltonians 
0125] or penalties associated with the strength of the 
set of qubits defining a logical qubit in the QUBO graph 
to hardware graph procedure |26j. 


In Sec. [TT] we present a strategy for tuning and opti¬ 
mizing a quantum annealing algorithm, finding the best 
parameters out of a pool of candidates and selecting the 
Hamiltonian specifications with the best performance. It 
is in this section where most of the new results are pre¬ 
sented. For accessibility to the readers, we divided this 
section into two main threads. Readers interested only 
in gauge selection (such as those researchers interested 
in benchmark studies, for example, on random spin-glass 
instances) can find the procedures needed in Sec. |II A 


- |II C| For readers interested in more general real-world 
application instances, where parameters for embedding 
procedures and other penalties need to be set, we devote 
Sec. to discussing the adjustments to the technique 
to deal with these additional challenges. In Sec. |III[ we 
delineate some future directions and possible further ap¬ 
plications of the present work. 


II. TUNING A QUANTUM ANNEALING 
ALGORITHM 


As previously discussed, there are many degrees of free¬ 
dom at the time of programming a quantum annealing 
device to solve a specific problem instance. Each realiza¬ 
tion of such degrees of freedom determine what we call a 
Hamiltonian specification for the quantum annealing cy¬ 
cles. For the purpose of generality, we leave the discus¬ 
sion at a very high-level form and in the following sections 
we will present application examples in different common 
practical scenarios, e.g., gauge selection and setting the 
strength of couplings among physical qubits representing 
a qubit from the original logical graph, also known as 
the embedding parameter setting problem [27] . In this 
general framework presented here, we only need to keep 
in mind that the performance of the device is determined 
by the programming degrees of freedom through the dif¬ 
ferent Hamiltonian specifications. The main question we 
discuss next: How do we select the Hamiltonian realiza¬ 
tion that yields the best performance of the device? It 
is the focus of this section to answer this question with 
a procedure requiring a minimum overhead, as described 
next. 


A. Performance Estimator: The Elite Mean, n e iit e 

Assume that for each Hamiltonian specification you 
can easily request a total number of readouts 7V rea d s 
from the quantum annealer. In some cases, these N rea ^ s 
must be obtained in batches due to programming limita¬ 
tions. For example, in the current D-Wave Two proces¬ 
sor hosted at NASA Ames, programming the device with 
an annealing time per cycle of t a = 100/is allows for a 
maximum number of readouts of 10,000. If the device is 
operated at t a = 20 /as this maximum number is 50,000 Q 
Therefore, while at t a = 20 /as a goal of 7V rea d s = 50, 000 
can be obtained in one shot, at t a = 100//S we need to re¬ 
quest 5 repetitions of 10,000 each. Let’s denote the num¬ 
ber of repetitions needed by n reps and therefore the num¬ 
ber of readouts in each repetition is n rea d s = A^ rea d s /^reps- 

For each readout, there is a corresponding 

£ising(s«) (Eq. 0 Let’s define by Ei S i ng as the ar¬ 
ray containing the n rea d s sorted energies, i.e., Ei S i ng = 
{ei, e 2 , • • • , e nreads } such that e^ < Cj for all j > i. Define 
^eiTte as the negative of the mean value of the lowest e 
percent of the energies in Ei S i ng . Since the array Ei S i ng 
contains n rea d s sorted energies from lowest to highest, 
then this expectation value is equivalent to calculating 
the mean value using the first n e ii te = [e * n rea d s / 100 ] 
values in Ei S i ng . Formally defined, 


^elite 

^elite(Treads) = — e i ( 2 ) 

i= 1 

Since only a fixed percent of the lowest energy values 
are included in the calculation, we refer to this score func¬ 
tion hereafter as the elite mean. This expression can be 
generalized to the case where several repetitions are used 
to collect the desirable total number of samples N vea & s 
by defining 


n 


e% 

elite 


(A"reads 5 ^ 


reps ) 


1 


^reps 



breads 

^reps 



(3) 


The minus sign in the definition Eq. U gives <ni e the 
interpretation of a score function or a performance esti¬ 
mator; the higher its value, the better the expected per¬ 
formance. Suppose one has several quantum annealers 
or several Hamiltonian specifications to choose from. If 
one is interested in assessing the performance of the de¬ 
vice with a number of reads N rea( \ s <C R. 99 (where i? 99 
is defined as the number of readouts needed to find the 
desired solution at least once with a 99% probability), we 
will show that n^ Pte serves as an effective score function 


1 In our machine, there is a maximum duty time per submission. 
This value is set to 10 6 /i,s, the reason why the maximum number 
of readouts at t a = 20 as is 50, 000 while only 10, 000 at t a = 
100 [is 
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or performance estimator that can be used to rank and 
to select the best of available quantum annealing spec¬ 
ifications to solve the problem at hand. The intuition 
for this score function follows from what is expected of 
a quantum annealing device: when given a problem to 
be solved, the quantum annealers (or the Hamiltonian 
specifications) that give the lower energy solutions are 
preferable, since a quantum annealer is designed to sam¬ 
ple from the lowest energy configurations. Therefore, the 
quantum annealer specification with the lower elite mean 
energy (or higher elite-mean score n^° e ), will give better 
performance. 


B. Performance Rank 


their frequencies. The process continue until ties are bro¬ 
ken, providing winners that are accordingly ranked lower. 
This method allows us to assign a unique ranking to any 
Hamiltonian specification whether or not we measured 
any ground states. Notice that in the particular case 
where the ground state is obtained for all the Hamilto¬ 
nian configurations explored, the performance rank will 
still assign lower ranks to Hamiltonian specifications with 
larger values ofp s , as desired, while breaking any ties that 
exist. 


C. Gauge selection: Case study with a random 
spin-glass instance 


In quantum annealing, the most natural gold standard 
for assessing performance is the probability of observing 
the ground state, since it translates into the probabil¬ 
ity of finding the optimal solution to the optimization 
problem studied. In more precise terms, let’s define the 
success probability of our quantum annealing algorithms 
by p s = ^gs/^Vtotah where n gs corresponds to the num¬ 
ber of observed ground states in the total number of re¬ 
quested readouts TVtotai- Given the number of rep¬ 
etitions needed to observe the optimal solution at least 
once with a 99% probability, R, 99 is given by m, 


R . 99 — 


ln(l - 0.99) 
ln(l -p s ) 


(4) 


The instances that will benefit the most from our se¬ 
lection approach are those hard-to-solve instances with 
a very low probability of obtaining the ground state, 
say with an i? 99 in the order of hundreds of millions 
or hundr eds of millions like the example discussed in 
Sec. IID| (R 99 A^ rea ds). The purpose of this work is to 
show a correlation of the performance estimator and the 
real performance of the machine. But how do we define 
or assess real performance when the number of ground 
states is not reasonably attainable for all the Hamilto¬ 
nian configurations we explored? Take for example the 
instance in Sec. m The default setting of the device 
does not provide even a single ground state solution af¬ 
ter A^otai = 50 x 10 6 readouts! To calculate p s for all 
gauges we would need to run for all 100 gauges being 
considered at least Af tota i > 50 x 10 6 , which is beyond 
the scope of this work. In this work we explore two def¬ 
initions of performance that allow us to rank different 
Hamiltonian configurations even in the case where the 
ground state is not obtained after a significantly large 
A^totai- The first and natural criteria is a greedy-like per¬ 
formance rank. This method gives a lower (better) rank 
to a Hamiltonian specification with a lower energy. In the 
common case of ties, they are broken by looking at the 
frequency (number of occurrences) of their lowest energy 
state. In case these are the same, the next lowest energy 
is compared and if they are still the same, one compares 


Benchmark studies assessing the presence or absence 
of speed-up of quantum annealers compared to classical 
processors resort to gauge selection as a way of obtain¬ 
ing reliable averaged results of the performance of the 
device d uni eh nu. Although it is known that gauge 
specification can significantly enhance the performance 
of the device, previous studies are limited to the scal¬ 
ing of the typical gauge since there is no a priori way 
to determine the optimal gauge. Gauge specification is a 
particular example of Hamiltonian specification discussed 
above. We present here how our performance estimator 
n^ te can be used to select the optimal gauges. To il¬ 
lustrate the procedure, we used a hard-instance out of a 
pool of random-spin glass instances similar to the ones re¬ 
ported elsewhere m ■ This instance was provided by Ser¬ 
gio Boixo, who assessed that this particular instance had 
a simulated-annealing (SA) runtime of the order of hun¬ 
dred times longer than the median instance, from a pool 
of hundreds of thousands of instances within the same 
family (instances with random couplings Rj G {+1,-1}, 
with 509 qubits as shown in Fig. [4|. As an abbrevia¬ 
tion, we refer hereafter to this specific random-spin glass 
example as the RS instance. For QA this instance was 
shown to also be particularly hard after not obtaining any 
ground states after trying 16 gauges, with 10,000 read¬ 
outs each, at 20 /is. Fig. [ljb) corroborates this assess¬ 
ment; the median gauge over a set of 100 random gauges 
has a p s = 1/(2 x 10 6 ) = 5 x 10 -7 , resulting in a expected 
number of repetitions to solution of R 99 ~ 9.2 x 10 6 an¬ 
nealing cycles. Since R, 99 is about two hundred times 
greater than N rea( ^ s = 50, 000, applying our performance 
estimator to select the optimal gauge before engaging 
in lengthier runs is expected to significantly reduce the 
computational time. 

Fig. [lja) shows there is a strong correlation between 
the rank obtained with n^T° e with 7V rea d s = 50, 000 and 
Tireps = 1, compared to the greedy performance rank de¬ 
scribed above. The number of total readouts used to es¬ 
timate the performance rank is 2 million per gauge. The 
error bars correspond to the rank provided by the first 
and third quartile out of 40 different experiment each 
with A^ rea ds = 50,000 for each of the 100 random gauges. 
The middle point corresponds to the median of the set of 
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experiments. Fig.[ljb) shows the same data set but with 
the raw values for II^ e and also serves the purpose of 
showing the count in the number of ground states, illus¬ 
trating that gauge selection can have a significant impact 
in the device performance, an increase as much as one to 
two orders of magnitude (see also Fig. 0 d) and Fig. |2|b) 
reflecting how the gauge selection can influence n gs .) 

As expected, any performance estimator would be a 
noisy metric and not expected to have a 100 % correla¬ 
tion with the real performance from an extensive number 
of readouts N tota \ R. 99 . The RS section of Table [T] (up¬ 
per half) addresses this issue. Suppose one utilizes the 
following strategy. One decides to run 100 gauges with 
a fixed Areads for each gauge. From this starting data 
set, and while processing the readouts in the search of 
an optimal solution, one can easily calculate n^° te . Since 
Are ads R. 99 , it is unlikely that this initial batch of cal¬ 
culations would contain the optimal solution. The refine¬ 
ment we propose here consists of using the information 
of the IL™ calculated on-the-fly to select, for example, 
the top 5 gauges (gauges with the highest II^ te score) 
out of the 100 random gauges. Since the selected gauges 
are expected to have a better performance than the typ¬ 
ical or average gauge, only the selected ones are used to 
continue with the remaining runs A to tai ~ R. 99 until the 
desired solution is found. Given this strategy, Table [T] 
answer the question: what is the probability that the ab¬ 
solute top gauge (that is, rank ed nu mber 1 according to 
the performance rank in Sec. II B| ) is contained in this 
set of predicted top 5 gauges? What is the probability 
of one finding any of the top 2 gauges in the set of pre¬ 
dicted top 5 gauges? etc, etc. Notice that at the level of 
A r eads = 50,000 which is ^200 times less than R. 99 , one 
obtains a reasonably high ^75% probability of obtaining 
the top 1 gauge in the set top 5 gauges predicted by II^ e . 
Table [I] also addresses the question of the existence of an 
optimal value of e for the performance estimator. In all 
the examples considered here it seems to be the case that 
a value of e% = 1 % or 2 % is optimal, a non-trivial result, 
since one might think incorrectly that the greater the 
number of low energy states included in the calculation 
of n* e the better. This table also shows the expected 
increase in the probability of choosing the top gauges as 
A r eads becomes larger. Note also the inclusion in the 
table of a “Greedy” column for each case. This new met¬ 
ric was included because the use of the greedy method 
for the performance rank described in Section B begs 
the question of why one could not use an even simpler 
performance estimator consisting of the same greedy ap¬ 
proach applied here to the case of a small number A rea d s 
instead of A tota i. However, the table clearly shows that 
H-eUte * s consistently always as good as, if not much bet¬ 
ter than, the greedy performance estimator. This is not 
surprising since as expected the n^ e should be a more 
robust metric than the greedy approach and to be less 
sensitive to rare-event occurrences. A clear advantage of 
the U 2 J° te is that it provides a score function that can be 
used for other purposes as will be shown elsewhere [28] . 


The greedy approach as a score function is expected to 
be much flatter, due to this ranking relying heavily on 
the breaking of ties whenever there are only a few low 
energy states that many gauges reach). 


D. Iterative strategy for embedding parameter 
setting and gauge selection 

When solving real-world applications there are several 
additional subtleties to take into consideration. While in 
the case of the random spin-glass instances there is only 
one objective function appearing in the quantum anneal¬ 
ing implementation, Ai S i ng in Eq. [l] in instances derived 
from real-world applications A ising is obtained from an¬ 
other cost energy function Aqubo, a quadratic binary 
expression containing the logical qubits, {^logical} before 
these are embedded into the hardware qubits, {^hardware} 
appearing in Ai S i ng [7]. From a practical application per¬ 
spective, we are interested in the possibility of using our 
performance estimator to select the best Hamiltonian 
specification with the smallest R m 99, therefore reducing 
the computational time. 

Once the embedding problem is solved [26, 27, 29] and 
one has a mapping of each logical qubit s^i £ { ^logical } 
into a subset of {^hardware}? the first decision to be made 
is to select the strength of the coupling Je needed to 
keep the hardware spins representing a logical spin in 
alignment with each other, [j For each value of Je , and 
after requesting A rea d s , we can calculate the fractions of 
the A rea ds that do not violate any of these embedding 
constraints, i.e., with all the physical spins representing 
logical spins being properly aligned. These solutions are 
said to satisfy the strict embedding (SE) requirement. 
The right y- axis in Fig. [3](a) shows the fraction of solu¬ 
tions passing this requirement out of the total number of 
readouts A rea d s , denoted as /se (Je). 

Intuitively the magnitude of Je cannot be too weak 
since it will not achieve the goal of keeping the spins 
properly aligned (same readout value for each variable 
representing the same logical qubit). Having a large value 
Je will certainly help in increasing the probability of not 
having any misaligned spins but it cannot be too strong 
either, Je max(J^), since after dividing everything 
by Je to make all — 2 < hi < 2 and —1 < J#, J^ < 1, 
the original values hi and Jij will be well below the pre¬ 
cision level and the performance will be significantly af¬ 
fected by noise. Therefore, a sweet spot with an optimal 
value of Je is expected. From our experience, /se serves 
as a guide for selecting the region of interest, denoted 

here as J^ < Je < J|f, with /se (J%) ~ 0.05 an d J|f 

corresponding to the onset of the plateou region with 


2 For simplicity we assume that all these penalties Je enforcing 
the embedding are equal. Further fine tuning can be done by 
optimizing each parameter but this is beyond the scope of this 
work. 
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Random spin-glass Fault diagnosis application 



FIG. 1. Correlation between n e iite rank vs. performance rank:, (a-d) We refer to “one experiment” (a single experiment) as the ranking 
of the 100 gauges according to the values of II^° e . In each of the 40 (100) experiments for the RS (50M-DMF) problem instance, each of the 100 
gauges is scored and ranked according to n^° te , obtained from N rea d s = 50,000 per gauge. The median value of these experiments is shown as 
the middle point and the limits of the error bars correspond to the 25- and 75-percentile from these experiments at t a = 20fis. The number of 
readouts (N rea ds = 50, 000) used to calculate II^° te rank in both examples is at least 100 times less than the number of repetitions required to find 
at least one solution with a probability of 99%, (i.e., R .99 N rea ds)- (a,c) The Spearman coefficient (rho) shows a moderately strong correlation 
of our estimator rank and the gold-standard rank used to define the performance. (b,d) Shown are the total number of ground states, n gs , across 
all extensive runs from -Easing (Equbo and using majority voting), with iV total = 2 millions (AT total = 5 million) for the RS (50M-DMF) problem 
instance. It is worth mentioning that the case of no-gauge with a performance rank of 77 out of 100 in panel (c) not even a single ground state was 
obtained after 50 millions readouts. Therefore, in this example a very bad gauge with a rank ~100 could take a significantly large computational 
time. 


TABLE I. Fraction of experiments where the performance estimator Uly° te (N rea ds) predicted correctly the top 1, either the top 
1 or 2 (labeled Any Top 2), and any among the top 1, 2, or 3 (labeled Any Top 3), etc, within the set of 5 top-ranked gauges 
out of 100 random gauges, for the cases of the random-spin (RS) and the hardest fault diagnosis (50M-DMF) problem instances 
described in the main text. The number of experiments is 400, 200, 80, and 40 for N rea d s = 5, 000,10, 000, 25, 000 and 50, 000, 
respectively. Note the non-trivial dependence on the fraction included in the elite mean, considering values of e = 1,2,5, and 
10 %, and showing an intermediate optimal value around 1% or 2%. As explained in the text, the greedy approach corresponds 
to the (1/iV r eads)* 100-percentile, i.e., rank based on lowest energy obtained and breaking ties with the frequency of the lowest 
energy states). Annealing time per readout for all experiments was 20 fi s. 
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/se ~ /s| ax in the P lot /se vs. J E - The value /g^ ax can 
be easily obtained experimentally in one-shot by setting 
J E > 1, and one can use that value to search for 

For the purpose of our discussion we selected two in¬ 
stances from the fault diagnosis application published 
elsewhere [? . The first instance, referred hereafter as 
300K-DMF, was selected because despite its implemen¬ 
tation with only 81 hardware qubits, it is unusually harcQ 
when compared with other benchmark studies HD], yet 
has a success probability just high enough to allow for 
a sizable number of ground states even in the worst 
Hamiltonian specification^] within a reasonable number 
of readouts set to /V rea d s = 300, 000 per gauge or J E con¬ 
sidered. As shown in Fig. [ 2 ] a finite number of ground 
states for every single gauge and every value of J E allows 


3 For the 300K-DMF, median success probability of p s = 
129/300, 000 = 4.3/10, 000 out of set of 100 random gauges 

4 For the 300K-DMF, the smallest p s = 25/300, 000 = 8.3/100, 000 


to rank each of these Hamiltonian specifications by their 
gold-standard performance rank and to compare with the 
rank predicted from our performance estimator. To sat¬ 
isfy the condition R ,99 /V rea d s , the n^ e per gauges 

was calculated by using only /V rea d s = 100. Fig. [2] shows 
that even with so few readouts, there is a strong corre¬ 
lation between our n e i ite score function and the number 
of ground states observed after /V to tai = 300, 000. Com¬ 
pared to the harder instances where 7V rea d s = 50,000, 
here we used an e% of 5% instead of 2%, since the latter 
would amount to computing the elite mean with only the 
two lowest values out of /V rea d s = 100. This makes the 
estimator too noisy and flat, analogous to the “Greedy” 
performance estimator that only uses the lowest value to 
rank gauges. As shown in Fig. [ 2 ] calculating the elite 
mean over the five lowest energies already gives a good 
correlation with n gs . 

The second instance, referred hereafter as 50M-DMF 
instance, serves the purpose of showing how our perfor¬ 
mance estimator can be used in a practical situation, 
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(a) J E scan with no gauge 


(b) Gauge scan with J E = 0.95 
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(c) J E scan with Gauge 85 



FIG. 2. The plots showing the strong correlation between our score function, Il^g, and number of ground states, n gs , for the 300K-DMF 
problem instance. The number of readouts, A/reads = 100, used to compute the score function is at least two orders of magnitudes less than the 
estimation of the typical number of readouts needed to observe the optimal solution at least once with a probability of 99% (77.99 = 10, 707 for 
the median performance over 100 random gauges). The error bars in 11^^° e (blue) corresponds to the 25-percentile and the 75-percentile from 500 
experimental realizations. The error bars in n gs (red) corresponds to the lowest and highest of three realizations, each with iV total = 100, 000, with 
the middle value as the median of the three experimental realizations. This shows that in these hard-instances, n gs is still a noisy value, but still 
our performance estimator places us in the range of the top gauges with the largest n gs . Each experimental realization consists of the estimation 
of n^J te (n gs ) by using 7V rea ds (A/total) per gauge. The iterative approach described here is the same one described in the text and in Fig.Jsjto set 
the embedding parameter Je and to select the optimal gauges. 


for instances with probabilities much smaller. These are 
the instances we expect to surface in the next genera¬ 
tion of quantum annealers. The instance 50M-DMF has 
the property of having a unique optimal solution, mak¬ 
ing it the most difficult to solve among the family of 
problem instances with six-faults to be diagnosed. More 
specifically, although the number of hardware qubits (96 
qubits) required to implement this instance is not un¬ 
usually large, this instance turned out to be extremely 
difficult for QA; not even a single-ground state was mea¬ 
sured after N tota \ = 50 x 10 6 annealing cycles, even af¬ 
ter optimizing for the optimal Je but under the default 
no-gauge! This instance was in large the motivation for 
defining a quick strategy to find the optimal Hamiltonian 
specifications (best Je and best gauge) capable of finding 
the ground state). 

Fig-i describes the suggested iterative approach used 
to optimize both the value of Je and to select the op¬ 
timal gauges in instances requiring a direct embedding 
approach. Starting with the no-gauge one can scan for 
the candidate values of Je and select the value of Je with 
the highest score n^ e . In contrast with the case of the 
RS instance above, here we cannot use to calculate 

the score function since, for example, the lowest energies 
of Rising will be different for every value of Je (because of 
the energy renormalization to fit all programmable values 
hi , J^j, and Je within the dynamical range |/q| < 2 and 
\Jij\ < 1). To circumvent this issue we compute n^° e 
after error-correcting the 7V rea ds solutions with majority 
voting [23 when going from {s hardware } ->■ {^logical}, and 
then sorting the states according to ^qubo ({^logical}) be¬ 
fore selecting the 2 % of the lowest energies used in the 
computation of n^° e . 

The next step in the procedure is to perform a gauge 
scan at the selected value of Je from the Je scans. 
Considering that we are dealing with instances with 
R, 99 A^ rea ds, it is not a significant usage of compu- 


J E scan with no gauge Gauge scan with J E = 0.85 



FIG. 3. Iterative strategy for parameter setting and gauge 
selection for the 50M-DMF instance: The corresponding data il¬ 
lustrates only one realization for the selection of Je (left) and another 
for the selection of the top 5 gauges (circled) with the highest score 
(right). For a statistical analysis of the robustness of the method, see 
the second part of Table [i] The left y-axis for the Je plots the value of 
the when calculated using 7V rea ds = 50, 000 which is more than 

three-orders of magnitude less than the number of readouts to solution, 
77.99, for this problem. The right y-axis for this Je plot shows the per¬ 
centage of solutions that passed strict embedding (solutions with no 
violations of the constraints imposed by Je)- /se serves as a guide for 
selecting the region with the optimal Je, as explained in the main text. 

tational resources of perform calculations with a number 
of gauges on the order of about 100. Notice that there 
is really no overhead while doing the gauge scans, since 
for every gauge considered, one needs to post-process all 
the solution readouts (e.g., with majority voting) while 
searching for the states with the optimal solutions any¬ 
ways. Since the energies of every single solution needs 
to be calculated, the only overhead in calculating n^° e 
comes from a cheap sorting of these energies before calcu¬ 
lating the elite mean. For NP-complete problems, we can 
always tell if we have found the desired answer. Also, for 
a large family of problems such as those NP-hard prob¬ 
lems where the NP-complete version is still interesting, 
one can still stop the search if the desired solution is ob¬ 
tained (e.g., we can ask whether there exists a solution 
with an energy lower than a reference energy, with the 
latter being for example the best solution attainable with 
a state of the art classical solver). Therefore, trying ^100 
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gauges in the search of an optimal gauge is not an unfea¬ 
sible idea. After calculating n^° e for the complete pool 
of gauge candidates, one can proceed to another set of 
J^-scans by using the best gauge with the highest score. 
As shown in Fig.[2|c) and from our experience with other 
problem instances where the procedure was even applied 
at different annealing times, in most of the cases the sec¬ 
ond optimal Je matched the same Je from the first scan 
under the no-gauge setting. Even in the cases where Je 
moved to a new value, the change was in the neighbor¬ 
hood of the first optimal value. As shown in Fig. §a) 
and (c), as long as one is near the optimal value of Je 
the performance is not significantly affected. The gauge 
selection, Fig. [2^b) , seems to have a much larger impact 
in the performance. Since the first Je ~scan does the job 
of taking us to the neighborhood of Je optimality (out 
of the set of candidates considered), it is reasonable to 
conclude that a second Je is not necessary and it is bet¬ 
ter to focus on the top gauges obtained from a gauge 
scan of 50-100 gauges. For easy instances a large gauge 
set would be unnecessary since the optimal solution will 
likely appear before one finishes going through the target 
number of 100 or so gauges. 

As shown in Fig. [l] the performance estimator pro¬ 
posed here is a noisy metric. For example, there is no 
guarantee that the top gauge is the same one as the one 
predicted by n^° e . Therefore, instead of selecting only 
the gauge predicted as top 1, it is advisable to select 
a handful of the predicted as top gauges as indicated 
in Fig[3jb). It is with this selected set that one per¬ 
forms the extensive runs 7V t otai ^ 77 gg, but where now 
R, gg has been significantly reduced given that we are run¬ 
ning with a set that includes the optimal gauge from the 
random set. Table [T] shows that selecting the predicted 
top five gauges has a high probability (> 80%) of con¬ 
taining the top 1 gauge yielding the largest number of 
ground states. Predicting any of the top 2 gauges had a 
probability ^90%. This is very remarkable considering 
that in this particular 50M-DMF problem running a low- 
performing gauge would lead to a significantly large time 
to solution. As mentioned above, the default no-gauge 
did not find the solution after 50 millions reads, there¬ 
fore yielding a 77gg > 230 millions, while any of the top 
3 gauges require 77.gg < 23 millions, providing at least 
an order of magnitude improvement in this hard-to-solve 
instance for the QA processor. 

In the case of real-world applications that use ancilla 
variables [a ns in the construction of their Fqubo post¬ 
processing strategies are also possible. In these cases it is 
more efficient to process the solution, for example, evalu¬ 
ating the problem energy, F? pro bi e m with only the relevant 
variables defining the problem. More specifically, and 
without loss of generality, we can express the set of result¬ 
ing logical qubits in the F^qubo expression as {^logical} 
= {^problem} U {^ancilla }5 where {s pro biem} corresponds 
to the set of qubits or binary variables that define com¬ 
pletely the problem description and that can be extracted 
to evaluate the energy of the problem, F pro bi e m- In these 


cases, and with the intention of increasing the chances of 
finding the optimal solution, it is more efficient to pro¬ 
cess the solution readouts with 7? pro biem({s pro biem}) and 
not with -Fqubo ({^logical}) • This postprocessing strategy 
can only help in finding the optimal solution, since for 
every readout F/ pro blem({^problem}) FzQUBO ({^logical}) 5 
therefore allowing for the possibiliy of finding optimal 
solutions in solutions that had been penalized by the an¬ 
cilla constrains. Our preliminary results indicate that for 
these problem instances, it is advisable to look also at the 
top 5 gauges obtained from the greedy approach (same 
approach described in Table .0 but now using F; pro biem 
instead of F’qubo) along with the top 5 from the n e i ite 
score function, also calculated with F? pro biem instead of 
F^qubo- The strategy proposed here consists of taking 
as the “selected top gauges” the union set of these two 
sets of top 5 gauges, and perform with these gauges the 
extensive runs with 7V t otai ~ R.9 g- 


III. CONCLUSIONS 

We defined a score function intended to estimate the 
performance of quantum annealers whose applicability 
does not rely on obtaining ground states corresponding 
to the desired solution. We observed a strong correlation 
of our performance estimator with the performance of 
the device even in the case where the number of readouts 
used to calculate it was several orders of magnitude less 
than the number of readouts required to find the desired 
solutions. The score function is based on a tail condi¬ 
tional expectation value, corresponding to the elite mean 
over a small percent representing the readouts with the 
lowest energies. We showed it can be used to efficiently 
select the optimal gauges from a large pool of random 
gauges and in setting Hamiltonian parameters appearing 
in the implementation of real-world applications. 

Although it has been previously shown that the de¬ 
cisions in programming quantum annealing devices can 
significantly impact the performance of the device cmi 
nu, thus far comparison of performance of quantum an¬ 
nealers to algorithms on conventional classical processors 
was limited to average performance over the selection of 
parameters explored. This study opens the possibility 
of revisiting such scaling studies, now with the oppor¬ 
tunity to select in advance the best configuration of the 
device. Having the possibility of selecting the specifica¬ 
tions (best gauges or other optimal parameters) will be 
indispensable once we start solving instances intrinsically 
harder with the new generation of quantum annealers. 
The overhead incurred to apply the selection procedure 
presented here is constant and it does not scale with the 
size of the system. We showed that even in cases where 
breads < 77.gg/1000, the method still works with a large 
probability of selecting the top gauges. 

In the case of real-world applications, the iterative 
strategy proposed in Sec. |IT d] requires essentially no over¬ 
head in calculating the performance estimator and used 




to rank the random gauges. Since the data needs to be 
processed anyway (e.g., calculation of E’qubo and ma¬ 
jority voting while testing whether or not the desired 
solution has been found), the only overhead incurred is 
the time needed to sort the solution before calculating 
the elite mean. In the case of other parameter settings 
such as the one used in the embedding problem, our per¬ 
formance estimator provides a very efficient approach by 
pinning down the region where the device has its best 
performance. 

Although our strategy allows us to select the best 
Hamiltonian specification in quantum annealers, we do 
not expect that it will be enough to change the com¬ 
plexity class seen in scaling studies mm Certainly, it 
could easily provide a speed-up of an order of magnitude 
from the default methods, as seen in some of the examples 
presented here, and it might be to-date the only feasible 
way to obtain solutions to hard-to-solve computational 
problems (either random spin-glass benchmarks or real- 
world applications) in the next generations of quantum 
annealers. 
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Appendix A: Chimera architecture Appendix B: Evidence against a commonly used 

rule-of-thumb for gauge selection 



FIG. 4. Device architecture and qubit connectivity D- 
wave Two at NASA Ames:. The array of superconducting 
quantum bits is arranged in 8 x 8 unit cells that consist of 8 
quantum bits each. Within a unit cell, each of the 4 qubits 
in the left-hand partition (LHP) connects to all 4 qubits in 
the right-hand partition (RHP), and vice versa. A qubit in 
the LHP (RHP) also connects to the corresponding qubit in 
the LHP (RHP) of the units cells above and below (to the 
left and right of) it. Edges between qubits represent couplers 
with programmable coupling strengths. Blue qubits indicate 
the 509 usable qubits, while grey qubits indicate the three 
unavailable ones out of the 512 qubit array. 


In the case of gauge selection, a commonly used ” rule- 
of-thumb” that had persisted in the community is that 
the gauge maximizing the number of antiferromagnetic 
couplings, Jij > 0 is preferred. The physical motivation 
behind this “rule” is that the precision in the specifi¬ 
cation of a > 0 (antiferromagnetic coupling) is more 
robust than its negative (ferromagnetic) counterpart 130:.. 
A more detailed analysis including 100 gauges for several 
problem applications considered (see Fig. [5]) shows that 
such rule-of-thumb does not hold in any of the hard in¬ 
stances considered here. Notice there is no correlation 
between the number of positive couplers and the perfor¬ 
mance of the specified gauge. We did no see any correla¬ 
tion either in any other quantity similar to J^: parame¬ 
ters studied include the number of Je > 0 (for the case of 
real-world applications with direct embedding), the num¬ 
ber of hi > 0 and the number of that are non- Je- For 
all those cases, still no correlation was found. 


Random spin-glass Fault diagnosis application 



FIG. 5. The low value of spearman correlation coefficient 
shows that there is basically no correlation between the num¬ 
ber of Jij > 0 resulting from a specified gauge and the per¬ 
formance in the device, ruling out the common belief that the 
larger the number of Jij > 0, the better. Shown here are 
examples from three different application domains. 
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